Deep Compositional Captioning: Describing Novel Object Categories without Paired Training Data

摘要

While recent deep neural network models have achieved promising results onthe image captioning task, they rely largely on the availability of corporawith paired image and sentence captions to describe objects in context. In thiswork, we propose the Deep Compositional Captioner (DCC) to address the task ofgenerating descriptions of novel objects which are not present in pairedimage-sentence datasets. Our method achieves this by leveraging large objectrecognition datasets and external text corpora and by transferring knowledgebetween semantically similar concepts. Current deep caption models can onlydescribe objects contained in paired image-sentence corpora, despite the factthat they are pre-trained with large object recognition datasets, namelyImageNet. In contrast, our model can compose sentences that describe novelobjects and their interactions with other objects. We demonstrate our model'sability to describe novel concepts by empirically evaluating its performance onMSCOCO and show qualitative results on ImageNet images of objects for which nopaired image-caption data exist. Further, we extend our approach to generatedescriptions of objects in video clips. Our results show that DCC has distinctadvantages over existing image and video captioning approaches for generatingdescriptions of new objects in context.